feat: dedup alerts so one failure = one incident#6
Merged
Conversation
Previously every alert opened an incident and fired an LLM call + Slack card, so a spiking error flooded the pipeline (N alerts -> N incidents). Add an in-process sliding-window gate in handle_alert: - fold repeat alerts for a still-open error signature into the live incident instead of opening a new one (the actual flood fix) - configurable burst threshold (SENTINEL_DEDUP_THRESHOLD, default 1) so a signature can be required to fire N times in the window before triaging - clear the signature at every terminal state (resolved / degraded / failed-and-reverted) so later alerts reopen normally - /alert skips background diagnostics for suppressed/duplicate alerts Also fixes a latent PK collision surfaced by the new tests: incident IDs were timestamped only to the second, so two same-second alerts collided and ON CONFLICT DO UPDATE overwrote the first. Added a random suffix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lushenwar
added a commit
that referenced
this pull request
Jul 14, 2026
The alert-dedup merge (#6) landed with two files that failed the `black --check --line-length 110` backend gate. Formatting only — no logic change; 20 core tests still pass. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Closes the noise/false-positive gap: previously every alert opened an incident and fired an LLM call + Slack card, so a spiking error meant N alerts → N incidents → N Claude calls → N Slack cards. This was the one interview answer the code genuinely didn't back.
Threshold-alone doesn't fix it (alert N+1 fires again after crossing the threshold). The real fix is open-incident dedup.
What
handle_alertnow runs a sliding-window gate:fire— open a new incident (default behavior for a fresh signature)duplicate— a still-open signature folds into the live incident, no new incident, no LLM/Slacksuppress— below the burst threshold (SENTINEL_DEDUP_THRESHOLD, default1= fire immediately; raise to require a burst withinSENTINEL_DEDUP_WINDOW_S, default 300s)Signatures clear at every terminal state (resolved / degraded / failed-and-reverted) so later alerts reopen normally.
/alertskips background diagnostics for suppressed/duplicate alerts.Also fixed
Latent PK collision the new tests surfaced: incident IDs were timestamped only to the second, so two same-second alerts collided and
ON CONFLICT DO UPDATEoverwrote the first record. Added a 2-byte random suffix.Scope notes
ponytail:with the upgrade path to Postgres/Redis if you go multi-worker.suppressed/duplicateshapes on/alert; stored incident schema and the/incidentscontract are unchanged, so the dashboard is unaffected.Tests
20 passed— addedtest_dedup_folds_repeat_alertsandtest_dedup_threshold_suppresses_below_burst.🤖 Generated with Claude Code